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Detailed Action 


Drawings 


Objections 

1 . The drawings are objected to because of the following. 

a. Referring to Fig. 2, the caption "CHECK WHETHER THERE EXISTS 
SOME OVERLAPPING AREA BETWEEN THE EXAMINED OLD FACE 
AND ONE OF THE SKIN REGIONS" is inconsistent with the Applicant's 
claimed invention, where a predetermined percentage of overlapping area 
is determined to exist. It should be clear that some overlapping area and a 
predetermined percentage of overlapping area is not the same. 

b. Referring to Fig. 2, the caption "SEARCH THE CORRESPONDIMG ... 
TECHNIQUE" should be changed to ""SEARCH THE CORRESPONDING 
... TECHNIQUE". 

c. Referring to Fig. 2, the caption "TURE FACE" should be changed to "TRUE 
FACE" 

A proposed drawing correction or corrected drawings are required in reply to the Office 
action to avoid abandonment of the application. The objection to the drawings will not 
be held in abeyance. 
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Specification 

Objections: Title of the Invention. 

2. The current title of the Applicant's invention, "System and Method for Rapidly 
Tacking Multiple Faces" (emphasis on placed on the word Tacking), should be replaced 
with the intended title, "System and Method for Rapidly Tracking Multiple Faces". 
Appropriate correction is required. 

Objections 

3. The disclosure is objected to because of the following informalities. The term 
"protrusion shape" (e.g. Applicant's specification page 7, line 19) is misleading. 
Protrusion means: the state of being protruded or projected outward, or something that 
is protruded or projected outward. The Applicant's usage of this word is not consistent 
with this definition ("contour of a face-like region ... having a height larger than its width 
([intuitively] similar to the symbol "IT') ... [, t]hat is, the face-like region has two sharp 
down-edges in its right and left sides" - Applicant's disclosure page 7, lines 19-21). 
There seems to be nothing in the Applicant's definition of "protrusion shape" that would 
necessarily protrude. It is requested that this terminology be changed throughout the 
disclosure (including the claims) to better reflect its definition. Appropriate correction is 
required. 
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Claims 


Objections 

4. Claims 1 , 10, 14, 16, and 17 are objected to because of the following 
informalities. These claims contain the word "tacking" where "tracking" was intended. 
Appropriate correction is required. 

5. Claim 16 is objected to because of the following informalities. The words 
"different person" should be replaced with "different people". Appropriate correction is 
required. 

6. Claims 9, 14 and 23 are objected to because of the following informalities. The 
word "existed" (e.g. Claim 9, line 4) should be replaced with "existing". Appropriate 
correction is required. 

7. Claim 13 is objected to because of the following informalities. The word "fault" 
should be changed to "false". Appropriate correction is required. 

Rejections Under 35 U.S.C. S 112(2) 

8. The following is a quotation of the second paragraph of 35 U.S.C. 112: 


9. Claims 1,5-8, 10-11, 13, 17 and 21-22 are rejected under 35 U.S.C. 112, second 
paragraph, as being indefinite for failing to particularly point out and distinctly claim the 
subject matter which applicant regards as the invention. 


The specification shall conclude with one or more claims particularly pointing out and 
distinctly claiming the subject matter which the applicant regards as his invention. 


t 
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10. The following is in regard to Claim 1 and 15. Referring to lines 21-23 of Claim 1, 
it is unclear whether the phrase "it is determined that the old face is still in the current 
frame and its position is in the center of the skin region" represents a 
functional/intended-use limitation or a limitation that sets forth substantive steps used in 
the Applicant's tracking system. In particular, one can conclude from the language of 
claim 1 that: "if yes" (i.e. if there exists more than a predefined percentage of overlapping 
area between the old face and a skin region), then "the old face is still in the current 
frame and its position is in the center of the skin region". Given this interpretation, the 
phrase denotes an observed or expected state of the "old face" when the condition "if 
yes" is true. The phrase, when interpreted in this manner, does not, in and of itself, lend 
structure to the claimed tracking system. Alternatively, the phrase may be interpreted 
such that: "if yes", then "the old face is still in the current frame" and its position is set to 
the center of the skin region. Notice that the latter interpretation results in a structural 
difference to the claimed tracking system (namely, a means to set the position of the old 
face to the skin region center) 1 . This interpretation is consistent with the Applicant's 
specification and will be adopted in the subsequent portions of this document. Claim 17 
suffers from the same ambiguity. 

1 1 . The following is in regard to Claim 5. It is unclear is meant by "variation of the 
face recorder". This will be taken to mean: variations in the tracked (old) face. 

12. The following is in regard to Claim 6 and 10. Referring to lines 3-4 of Claim 6, it is 
unclear whether the phrase "so as to separate connected face regions" represents a 
functional/intended-use limitation or a limitation that sets forth substantive steps used in 


Also notice that, given either interpretation, the phrase "the old face is still in the current frame" merely indicates the 
observed state of the old face and, in that regard, does not represent a structural limitation of the claimed tracking 
system. 
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the Applicant's tracking system. A similar issue exists in Claims 1 and 15 and was 
addressed above. Since this phrase does not seem to introduce any structure into the 
claimed tracking system, it will be treated as a functional limitation. 

1 3. The following is in regard to Claims 7 and 21. The functions e( ), p( ), and d( ) are 
undefined. The variables, ij and w are also undefined. Without defining these functions 
and variables, the condition set forth in claims 7 and 21 is rendered unclear. Its meaning 
is not apparent from mathematical expression alone. In order to impart meaning to this 
condition, it is advised that the Applicant provide descriptions of these functions and 
variables consistent with the specification. 

14. The following is in regard to Claim 1, 8, 13, 17, and 22. It is not clear what is 
meant by "true new faces". Specifically, it is not clear how or why certain new faces are 
characterized as being true. True new faces will interpreted, henceforth, as meaning: 
detected faces or candidate face regions that do not match or correspond to any of the 
tracked (old) faces (i.e. they are newly occurring faces), yet satisfy the geometric, color, 
and motion-based criteria for being a face. 

15. Claim 10 recites the limitation "based on the faces". There is insufficient 
antecedent basis for this limitation in the claim and it cannot be ascertained as to what 
faces the Applicant is referring to. These faces will be taken to mean tracked (old) faces. 

Rejections Under 35 U.S.C. S 103(a) 

16. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as 
set forth in section 102 of this title, if the differences between the subject matter sought to be 
patented and the prior art are such that the subject matter as a whole would have been obvious at 
the time the invention was made to a person having ordinary skill in the art to which said subject 
matter pertains. Patentability shall not be negatived by the manner in which the invention was 
made. 
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17. Claims 1-2 and 17-18 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Haritaoglu et al. ("Hydra: Multiple People Detection and Tracking Using 
Silhouettes", CVPR 1999), in view of Yang et al. ("Tracking Human Faces in Real- 
Time", Carnegie Mellon University Technical Report, CMU-CS-95-210, 1995). 

18. The following is in regard to Claims 1 and 17. Haritaoglu et al. disclose a real- 
time system for detecting and tracking multiple people (in particular their heads - 
Haritaoglu et al. Section 3.1) in sequences of images (Haritaoglu et al. Abstract). The 
system includes the following: 

(1 .a.) Generation of face-like regions (e.g. head regions - Haritaoglu et al. Fig. 
5) by: 

(1 .a.2) Analyzing motion (Haritaoglu et al. page 6, left column, last 
paragraph, sentences 1-3) to determine possible face-like 
regions (e.g. by estimating or predicting head locations - 
Haritaoglu et al. page 6, right column, lines 10-30) from 
foreground regions or blobs 2 (Haritaoglu et al. Abstract, 
sentence 3 and page 2, left column, lines 3-6) based on 
moving information of the input image. 
(1 .b.) Recording tracked faces (e.g. in a "list of heads to be tracked" - 

Haritaoglu et al. page 7, left column, last sentence). Also, note that this 


The foreground regions considered here are the portions of Haritaoglu et al.'s foreground regions that correspond to 
a head or face. Though Haritaoglu et al. do not distinguish these as being skin-colored regions, it should be readily 
understood that, when performing color segmentation such as Yang et al. (see below), these portions of the 
foreground would be classified as skin-colored regions. This does not diminish the validity or relevance of Haritaoglu 
et al.'s teachings or the discussion, herein, with regard to items (1.a)-(1.e). 
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(1.c.) 


d-d.) 


(I.e.) 


feature would be inherent to any object tracking method or system, since 
tracking objects requires a priori knowledge of the objects' previous 
position or state. 

Checking the foreground regions and the faces previously tracked and 
recorded (i.e. heads of the aforementioned "list of heads to be tracked") 
to determine whether the face-like regions are old faces which have 
been tracked in a previous frame (i.e. existing heads) or are possible new 
faces. See Haritaoglu et al. page 7, right column, last paragraph, 
sentences 3-4. 

Determining whether the possible new faces are true new faces 
(Haritaoglu et al. page 7, right column, last paragraph, sentences 3-4 and 


Tracking multiple faces (Haritaoglu et al. Figs. 2 and 4, and page 6, left 
column sentences 1-3) based on the new and old faces (Haritaoglu et al. 
page 7, right column, last paragraph, sentences 3-4 and 7-8). Clearly, this 
tracking is also based on the foreground regions. See, for example, 
Haritaoglu et al. Figs. 2 and 4, noting that the silhouettes depicted therein 
represent the foreground regions. See also Haritaoglu et al. page 2, left 
column, lines 3-6. This tracking involves: 

(1 .e.1 ) When a tracked face is a new face, the new face is recorded 
(Haritaoglu et al. page 7, left column, last paragraph, sentence 


(1 .e.2) When a tracked face is an old face, it is determined whether 
there exists more than a predefined percentage of overlapping 
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area (i.e. the bounding boxes of the old, tracked face 
sufficiently overlap those of the detected foreground region) 
between the old face and a skin region (i.e. the bounding box 
of the old, tracked face sufficiently overlaps those of the 
detected foreground region - Haritaoglu et al. page 7, left 
column, last paragraph, sentence 3 - as determined by the 
so-called occupancy test - Haritaoglu et al. page 6, left 
column sentences 1-2). 

1 . If there exists more than a predefined percentage of 
overlapping area between the old face and a skin region, 
the foreground region corresponds to tracked head 
(Haritaoglu et al. pages 6-7, section 4, paragraph 2). 
Clearly, the old face is still in the current frame. 
Furthermore, the position of the tracked head is set to the 
median (i.e. center) of the skin region (Haritaoglu et al. 
page 6, right column, lines 10-12). 

2. Otherwise, the position of the old face is determined by 
correlation operation (Haritaoglu et al. page 7, right 
column, lines 1-3). If the bounding boxes do not sufficiently 
overlap and, consequently, there is no match between any 
tracked head with the detected, the location of the old, 
tracked head is updated via the motion model. According 
to Haritaoglu et al. (Haritaoglu et al. page 6, lines 22-30), 
updating the motion model involves a correlation 
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operation. 

19. Haritaoglu et al. suggest performing the detection of colors and/or textures 
corresponding to skin (face) and generating or, more specifically, segmenting the 
foreground objects into skin regions corresponding to the heads of detected individuals 
(Haritaoglu et al. page 3, left column, Appearance Information), However, the face 
tracking system of Haritaoglu et al. does not make use of such information. 

20. Yang et al. disclose a real-time system for tracking human faces. This system 
involves: 

(1.a.1) Generating a plurality of skin regions by detecting skin color pixels of an 
input image (Yang et al. section 4.1, paragraph 1 and Fig. 10, all on page 
16). 

21 . The teachings of Yang et al. and Haritaoglu et al. are combinable because they 
are analogous art. Specifically, both Yang et al. and Haritaoglu et al. disclose systems 
and methods for detecting and tracking human faces. Therefore, given the teachings of 
Yang et al., it would have been obvious to one of ordinary skill in the art, at the time of 
the applicant's claimed invention, to further specify the individuals, or the heads of the 
individuals, tracked in the system of Haritaoglu et al., by generating a plurality of skin 
regions by detecting skin color pixels of an input image. The motivation to do so would 
have been to accommodate color sequences (as opposed to simply grayscale, as in 
Haritaoglu et al.'s system). Moreover, by exploiting the relatively distinct color attributes 
of human skin (Yang et al. Section 3.1), one could more accurately localize individuals 
and individuals' heads, thereby, facilitating detection and tracking. Similar arguments 
can be made for Claim 17. 

22. The following is in regard to Claim 2 and 18. As shown above, the teachings of 
Yang et al. and Haritaoglu et al. can be combined so as to yield a face tracking system 
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that adequately satisfies the limitations of claim 1 . According to the tracking methodology 
proposed by Haritaoglu et al. t detected head regions (e.g. skin regions obtained 
according to Yang et al.'s face location scheme) that sufficiently overlap with tracked 
heads (i.e. there exists more than a predefined percentage of overlapping area between 
an old face and a skin region) are determined to correspond to (i.e. are labeled as) a 
tracked (old) face (Haritaoglu et al. page 5, Section 4, 1 ret bullet, page 6 left column, lines 
1-8. and page 7, left column paragraph 1, sentences 3-4). Similar arguments can be 
made for Claim 18. 

23. Claims 3-5 and 19 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Haritaoglu et al., in view of Yang et al., in further view of Lee et al. (U.S. Patent 
6,542,625). 

24. The following is in regard to Claim 3 and 19. As shown above, the teachings of 
Yang et al. and Haritaoglu et al. can be combined so as to yield a face tracking system 
that adequately satisfies the limitations of claim 1. However, neither Haritaoglu et al. nor 
Yang et al. expressly show or suggest that: 

(3.a.) The luminance difference between two successive images is used as a 
moving information. 

(3.b.) A pixel is defined as a moving pixel if its luminance difference between two 

adjacent images is larger than a threshold. 
(3.c.) If there are more than a predefined percentage of pixels classified as 

moving pixels in a skin region, the region is labeled as a possible face-like 

region. 
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25. Lee et aL, on the other hand, disclose a disclose a method (Lee et al. Abstract 
and Field of Invention) for detecting a specified object (e.g. a face - Lee et al. column 2, 
lines 14-16) in a sequence of images, wherein: 

(3.a.) The luminance difference between two successive images is used as a 

moving information (Lee et al. column 4, lines 43-47) 
(3.b.) A pixel is defined as a moving pixel if its luminance difference between two 
adjacent images is larger than a threshold (Lee et al. column 4, lines 47- 
55). 

(3.c.) If there are more than a predefined percentage of pixels classified as 

moving pixels in a skin region 3 , the region is labeled as a possible face-like 
region (Lee et al. column 5, lines 4-46). This labeling is accomplish as 
follows. 

1 . Connected regions, designated with a binary value of 1 , are formed to 
indicate moving pixels in a skin region (Lee et al. column 5, lines 23-29 
and 31-34). 

2. Minimum bounding boxes (MBR - Lee et al. column 5, lines 35-37) of 
these connected regions are determined. 

3. If the regions enclosed in an MBR has a density of 1 pixels greater than 
a predetermined threshold (Lee et al. column 5, lines 37-40), they are 
classified as possible face-like regions (i.e. object candidate regions - 
Lee et al. column 5, lines 37-38). This density essentially indicates the 


These pixels include, for example, those that are skin colored [Lee et al. Fig. 3A, step 305] and undergo movement 
[Lee et al. Fig. 3A t step 306]. Such pixels are detected by applying an AND operation in the manner discussed in 
Lee et al. column 5, lines 4-21 . Thus, moving pixels in color regions consist of those pixels presumably having a 
binary value of 1 after the performing the AND operation. This can be inferred from the subsequent description of 
steps 307-308 of Lee et al. Fig. 1, discussed in Lee et al. column 5, lines 25-34. 
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percentage of moving pixels in a skin region. 

26. The teachings of Lee et al. are combinable with those of Haritaoglu et al. and 
Yang et al. because they are analogous art. Specifically, these teachings are all directed 
toward detecting and tracking faces in a series of images. Therefore, given the teachings 
of Lee et al., it would have been obvious to one of ordinary skill in the art, at the time of 
the applicant's claimed invention, to integrate steps (3.a)-(3.c) above into the facial 
tracking system obtained by combining the teachings of Haritaoglu et al. and Yang et al., 
in the manner discussed above. The motivation to do so would have been to effectively 
detect objects that move sufficiently in adjacent frames and to remove from 
consideration regions that do not correspond to the expected characteristics of the 
human face. This would, in turn, reduce the computational load of a system thus 
obtained, by limiting the searched domain of the image to regions where a face is likely 
to be. Similar arguments can be made for Claim 19. 

27. The following is in regard to Claim 4. As shown above, the teachings of Yang et 
al., Haritaoglu et al. and Lee et al. can be combined so as to yield a face tracking system 
that adequately satisfies the limitations of claim 3. The skin-color model used by Yang et 
al. accommodates a variety of different skin colors (e.g. Yang et al. page 1 1 , lines 1-8 
and Fig. 6). Furthermore, Yang et al. demonstrate the usage of an adaptive skin color 
model representing skin features of different people to generate a plurality of connected 
skin regions (Yang et al. Section 3.2: Skin-Color Adaptation). Given these teachings, it 
would have been obvious to one of ordinary skill in the art, at the time of the applicant's 
claimed invention to use an adaptive color model to generate skin regions. The 
advantage of using an adaptive skin color model, such as Yang et al.'s, in a facial 
tracking system would have been to make such a system more robust by allowing a 
more generalized detection of people, including people of different races (Yang et al. 
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page 1 1 , lines 6-7). Moreover, adaptive color models are more resilient to variable 
lighting conditions and viewing parameters (Yang et al. Section 3.2, paragraph 1). 

28. The following is in regard to Claim 5*. As shown above, the teachings of Yang et 
al., Haritaoglu et al. and Lee et al. can be combined so as to yield a face tracking system 
that adequately satisfies the limitations of claim 4. According to Yang et al. (Yang et al. 
Section 3.2: Skin-Color Adaptation), the adaptive color model is responsive to variations 
in the tracked face, and its parameters (e.g. weights a, p, and y in (EQ 8)-(EQ 10) of 
Yang et al.) are updated accordingly. Indeed, this represents the whole notion of an 
adaptive color model - that is, model parameters are adapted dynamically in response 
to a priori image information. For example, the weights a, p, and y in (EQ 8)-(EQ 10) of 
Yang et al. encapsulate prior facial-region information that is used to influence current 
parameters (Yang et al. page 14, last paragraph, sentence 1). 

29. Claims 6 and 20 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Haritaoglu et al., in view of Yang et al. and Lee et al., in further view of Kinjo (U.S. Patent 
6,529,630). 

30. The following is in regard to Claim 6 and 2(f. As shown above, the teachings of 
Yang et al., Haritaoglu et al. and Lee et al. can be combined so as to yield a face 
tracking system that adequately satisfies the limitations of claim 5. Though the face 
tracking systems of Yang et al. and Haritaoglu et al. both involve the generation and 


Recall the manner in which the phrase "variation of the face recorder" is being interpreted. See the discussion above 
regarding the U.S.C. § 112(2) rejections of the corresponding claims. 

Recall the manner in which these claims are being treated in this document. See the discussion above regarding the 
U.S.C. § 112(2) rejections of the corresponding claims. 
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subsequent analysis of "silhouettes" of a tracked faces (e.g. Yang et al. Fig. 10(b) or 
Haritaoglu et al. Figs. 4-5), neither Haritaoglu et al., Yang et al., nor Lee et al. expressly 
show or suggest an analysis of a silhouette that involves determining whether there 
exists a "protrusion shape" in the image so as to separate connected face regions. 
31 . Kinjo discloses a method and device for extracting principal subjects (especially 
human faces - Kinjo Fig. 4) from input digital images (Kinjo Abstract). The extraction of 
human faces involves analyzing (in the face contour/circular shape extracting section 80 
shown in Kinjo Fig. 4) the geometric properties of potential face contours (i.e. 
"silhouettes"), obtained by binarizing and differentially filtering an input image 6 (Kinjo 
column 10, lines 66-67 to column 1 1 , lines 1-3). Then, it is determined whether there 
exists silhouettes having a shape that, inter alia, includes lines (i.e. specified lines - 
Kinjo column 11, line 9), corresponding to the sides of a face, having a length and 
distance of separation, consistent with a typical human face (Kinjo column 1 1 , lines 4- 
18). It is further determined whether the shape includes lines perpendicular to the 
specified lines that correspond to the top and bottom of a face (Kinjo column 11, lines 
19-30). Such a shape can be reasonably considered a "protrusion shape". This 
interpretation is consistent with the Applicant's definition of a "protrusion shape" stated 
on page 7 of the Applicant's specification, because it represents a "contour of a face-like 
region ... having a height larger than its width ([intuitively] similar to the symbol "n") ... [, 
t]hat is, the face-like region has two sharp down-edges in its right and left sides". Skin- 
color regions (Kinjo column 10. lines 36-44) determined to have such a shape are 
designated as face candidate regions (Kinjo column 11, lines 36-42). 


It is well known that differentially filtering an image (e.g. by a Sobel operator) generally produces a filtered image 
where edges between regions of sufficient contrast are enhanced, while regions separated by these edges are 
suppressed. In the case of face, these edges effectively delineate the face, thereby, producing what resembles a 
facial silhouette or outline. 
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32. The teachings of Kinjo are combinable with those of Haritaoglu et al., Yang et al., 
and Lee et al. because they are analogous art. Specifically, Haritaoglu et al., Yang et al., 
Lee et al., and Kinjo all disclose systems and methods that detect regions of an image 
corresponding to a human face that satisfy certain geometric constraints. Therefore, it 
would have been obvious to one of ordinary skill in the art, at the time of the applicant's 
claimed invention, to detect potential face regions during face tracking by further 
imposing constraints on the geometry of the these regions, as Kinjo does. The 
motivation to do so would be to further limit the number of candidate regions that will be 
tracked, or otherwise undergo further processing, by eliminating regions that do not 
adhere to the typical geometry of a human face. This, in turn, mitigates the 
computational load on the system and, moreover, makes the tracking system more 
robust by reducing the potential for the erroneous tracking of skin-color regions that may 
not correspond to a human face. Similar arguments can be made for Claim 20 

33. Claims 8 and 22 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Haritaoglu et al., in view of Yang et al., Lee et al., and Kinjo, in further view Turk et al. 
("Eigenfaces for Recognition"). 

34. The following is in regard to Claim 8 and 22 7 . As shown above, the teachings of 
Yang et al., Haritaoglu et al., Lee et al. and Kinjo can be combined so as to yield a face 
tracking system that adequately satisfies the limitations of claim 6. As discussed 
extensively above, Yang et al., Haritaoglu et al., Lee et al. and Kinjo each remove from 
consideration (i.e. "filter out") those candidate face regions (i.e. possible new faces) that 

7 Recall the manner in which these claims are being treated in this document. See the discussion above regarding the 
U.S.C. § 112(2) rejections of the corresponding claims. 
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represent detection failures or otherwise fail to adequately satisfy the various geometric, 
color (these "filtered" regions can be considered "false faces"), and/or motion-based 
criteria set forth in their respective disclosures (e.g. Yang et al. Section 4.4 paragraph 1, 
Haritaoglu et al. page 7, left column, last two sentences, Lee et al. column 4, lines 43-60, 
Kinjo column 1 1 , lines 37-42). Though Yang et al. indicate that eigenface analysis can 
be used for facial recognition (Yang et al. page 2, sentence 3), neither Yang et al., 
Haritaoglu et al., Lee et al. nor Kinjo expressly show or suggest that the other possible 
new faces are verified as true new faces by eigen-face analysis. 
35. Eigenfaces is a well-known method for recognizing a face by comparing it to 
faces of known individuals. The concept of eigenfaces was first introduced by Turk et al. 
in the reference cited above. Turk et al. propose a real-time system that locates and 
tracks a subject's face. The eigenfaces are used to recognize the tracked face as one a 
set of known faces. (Turk et al. Abstract and Fig. 5). Specifically, Turk et al. detect and 
track regions (i.e. blobs or subimages - Turk et al. page 77, left column, paragraphs 2-3) 
of the input sequence of images that likely correspond to a face. By projecting each of 
these newly detected regions into a face-space (Summary of Eigenface Recognition 
Procedure steps 5-6 on Turk et al. page 76 and paragraph 3 of the section entitled Using 
Eigenfaces to Classify a Face Image on page 75) spanned by the set of eigenfaces 
(paragraph 2 of the section entitled Calculating Eigenfaces on Turk et al. pages 73-74), 
these regions are recognized as either known (e.g. previously tracked) faces (steps 5-6 
of the section entitled Summary of Eigenface Recognition Procedure on page 76 of Turk 
et al.) - or "true new faces" 7 (i.e. unknown faces - Learning to Recognize New Faces on 
page 79 of Turk et al. and step 5 of the section entitled Summary of Eigenface 
Recognition Procedure on page 76). 
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36. The teachings of Turk et al. are combinable with those of Yang et al., Haritaoglu 
et al., Lee et al. and Kinjo because they are analogous art. In particular, the teachings of 
Yang et al., Haritaoglu et al., Lee et al., Kinjo, and Turk et al. are all directed toward face 
detection and/or tracking. Therefore, given the teachings of Turk et al., it would have 
been obvious to one of ordinary skill in the art, at the time of the applicant's claimed 
invention, to determine whether faces are truly new (unknown) faces or known faces, 
presumably those that have been previously tracked. The motivation for verifying 
whether candidate face regions correspond to truly new faces or to previously tracked 
faces would have been to allow the tracking of faces of individuals who have just entered 
a scene, while continuing to track the faces of tracked individuals. The former is 
suggested by Turk et al. (Turk et al. page 79, Learning to Recognize New Faces). The 
motivation to use eigenfaces to accomplish this would have been to exploit its well- 
accepted simplicity and efficiency, as well as its ability to learn and later recognize new 
faces in an unsupervised manner (Turk et al. Abstract, last sentence). Similar arguments 
can be made for Claim 22. 

37. Claims 9 and 23 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Haritaoglu et al., Yang et al., Lee et al., Kinjo, and Turk et al., as applied to claims 8 and 
22 above, in further view of Abdel-Mottaleb et al. (U.S. Patent 6,263,1 13). 

38. The following is in regard to Claim 9 and 23. As shown above, the teachings of 
Yang et al., Haritaoglu et al., Lee et al., Kinjo, and Turk et al. can be combined so as to 
yield a face tracking system that adequately satisfies the limitations of claim 8. 
Furthermore, Kinjo and Lee et al. suggests that candidate face regions should be 
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removed from consideration (i.e. "filtered out" and/or designated as "false faces") if the 
following attributes fail to satisfy certain, predetermined criteria: 

(9.a.) Compactness of a tracked region (i.e. the density of the grids - Lee et al. 

column 5, lines 37-40). 
(9.b.) Ratio between the height and width of a face (i.e. the aspect ratio - Kinjo 

column 11, lines 32-35) 
(9.c.) Number of holes existing in a region (Kinjo column 14, lines 18-24). The 
position and geometry of the eyes are used as internal structural 
characteristics when designating a region as a candidate face region (Kinjo 
column 14, lines 24-25). Specifically, a region is designated as a candidate 
face region if the (two) eyes appear in the expected positions within a face 
(Kinjo column 14, lines 20-24). The regions corresponding to the eyes 
appear, in the binarized image, (Kinjo column 13 lines 66-67 to column 14, 
line 1) as black regions with respect to white regions corresponding to skin, 
etc. In this manner, the eyes can be interpreted as representing "holes" in 
the skin region. Adopting this language, one can infer from Kinjo's 
teachings that if less than two holes (eyes) appear in a region it will not be 
designated as a candidate face region, that is, it represents a "false face". 
(9.d.) Convexity of a face (Kinjo, column 1 1 , lines 4-7). 
Though Kinjo suggest that an operator execute the steps discussed in (9.c), this process 
is clearly amenable to automation. Therefore, it would have been obvious to one of 
ordinary skill in the art, at the time of the applicant's claimed invention, to automate this 
process in order to provide real-time, unsupervised face detection. Despite this, neither 
Yang et al., Haritaoglu et al., Lee et al., Kinjo, nor Turk et al. show or suggest using the 
(9.e.) statistical variance of a potential face to "filter out false faces". 
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39. Abdel-Mottaleb et al. disclose a method for detecting a face in an image, wherein 
the statistical variance of a potential face (e.g. a skin colored region - Abdel-Mottaleb et 
al. column 3, lines 31-38) to "filter out" (i.e. remove from consideration) "false faces" 
(Abdel-Mottaleb et al. column 4, lines 14-17). 

40. The teachings of Abdel-Mottaleb et al. are combinable with those of Yang et al., 
Haritaoglu et al., Lee et al., Kinjo, and Turk because they are analogous art. In 
particular, Yang et al., Haritaoglu et al., Lee et al., Kinjo, Turk and Abdel-Mottaleb all 
disclose systems involving the detection of a face within an image. Therefore, it would 
have been obvious to one of ordinary skill in the art, at the time of the applicants claimed 
invention, to use the statistical variance to "filter out" candidate faces that should not be 
further considered (i.e. "false faces"). The motivation to do so would have been to 
eliminate extraneous regions from tracking or other further processing. Since human 
faces typically have a relatively uniform color, regions having highly varying color should 
typically not be considered. Similar arguments can be made for Claim 23. 

41. Claims 10-12 and 16 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Haritaoglu et al., in view of Yang et al., in further view Kinjo. 

42. The following is in regard to Claim 10. As discussed above with regard to Claim 
1 , Haritaoglu et al. and Yang et al., when combined in the manner suggested above, 
yields a system for tracking multiple faces comprising: 

(10.a.) Generation of face-like regions (e.g. head regions - Haritaoglu et al. Fig. 


5) by: 


(10.a.1 ) Generating a plurality of skin regions by detecting skin color 
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pixels of an input image (Yang et al. section 4.1 , paragraph 
1 and Fig. 10, all on page 16). 
(10.a.2) Analyzing motion (Haritaoglu et al. page 6, left column, last 
paragraph, sentences 1-3) to determine possible face-like 
regions (e.g. by estimating or predicting head locations - 
Haritaoglu et al. page 6, right column, lines 10-30) from skin- 
colored regions (Haritaoglu et al. Abstract, sentence 3 and 
page 2, left column, lines 3-6) based on moving information 
of the input image. 

(10.b.) Determining whether the possible faces are new faces (Haritaoglu et al. 

page 7, right column, last paragraph, sentences 3-4 and 7-8). 
(10.c.) Tracking multiple faces (Haritaoglu et al. Figs. 2 and 4, and page 6, left 
column sentences 1-3) based on the faces (Haritaoglu et al. page 7, 
right column, last paragraph, sentences 3-4 and 7-8) and skin-colored 
regions (Yang et al. Section 4.1). 
43. It was also shown above that Kinjo teaches: 

(1 0.a.3) analyzing a silhouette to determine whether there exists a protrusion 
shape in the image so as to separate connected regions. 
The details will be omitted here. The reader is directed to the discussion above 
regarding claim 6. Following a similar line of reasoning, as presented there, it would 
have been obvious to one of ordinary skill in the art, at the time of the applicant's claimed 
invention, to detect potential face regions during face tracking by further imposing 
constraints on the geometry of the these regions, as Kinjo does. The motivation to do so 
would be to further limit the number of candidate regions that will be tracked, or 
otherwise undergo further processing, by eliminating regions that do not adhere to the 
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typical geometry of a human face. This, in turn, mitigates the computational load on the 
system and, moreover, makes the tracking system more robust by reducing the potential 
for the erroneous tracking of skin-color regions that may not correspond to a human 


44. The following is in regard to Claim 1 1. As shown above, the teachings of Yang et 
al., Haritaoglu et al., and Kinjo, can be combined so as to yield a face tracking system 
that adequately satisfies the limitations of claim 10. Furthermore, as shown above with 
respect to claim 1, Haritaoglu et al. teaches: 

(1 1 .a.) Recording faces that have been tracked (e.g. in a "list of heads to be 
tracked" - Haritaoglu et al. page 7, left column, last sentence). 

(1 1 .b.) Checking the face-like regions and the faces previously tracked and 

recorded (i.e. heads of the aforementioned "list of heads to be tracked") 
to determine whether the face-like regions are old faces which have 
been tracked in a previous frame (i.e. existing heads) or are possible 
new faces. See Haritaoglu et al. page 7, right column, last paragraph, 
sentences 3-4. 

The system of Haritaoglu et al. tracks face-like regions that correspond 
to previously tracked faces in the normal fashion. Face-like regions not 
corresponding to a previously tracked face (i.e. possibly newly detected 
faces) are treated differently. These regions must be verified to 
determine whether they are truly new faces or detection errors. See 
Haritaoglu et al. page 7, left column, last paragraph. 

45. The following is in regard to Claim 12. As shown above, the teachings of Yang et 
al., Haritaoglu et al., and Kinjo, can be combined so as to yield a face tracking system 
that adequately satisfies the limitations of claim 1 1 . Furthermore, as shown above with 


face. 


Application/Control Wmber: 09/931 ,237 


Art Unit: 2623 


Page 


respect to claim 1, Haritaoglu et al. teaches that if it is determined that there exists more 
than a predefined percentage of overlapping area between a face-like region and a 
tracked face, the face-like region is labeled as a tracked old face. See the discussion 
above with regard to item (1.e.2) of claim 1. 

46. The following is in regard to Claim 16. As shown above, the teachings of Yang et 
al., Haritaoglu et al., and Kinjo, can be combined so as to yield a face tracking system 
that adequately satisfies the limitations of claim 1 1 . Furthermore, as shown above with 
respect to claim 4, Yang et al. teaches generating a plurality of connected skin regions 
based on an adaptive skin color model representing skin features of different people 
(e.g. different races). The details will be omitted here. Refer to the discussion above 
relating to claim 4. 

47. Claims 13-15 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Haritaoglu et al., Yang et al. and Kinjo, as applied to claim 1 1 , in further view of Turk et 
al. 

48. The following is in regard to Claim 13. As shown above, the teachings of Yang et 
al., Haritaoglu et al., and Kinjo, can be combined so as to yield a face tracking system 
that adequately satisfies the limitations of claim 1 1 . As shown above, with respect to 
claim 8, given the teachings of Turk et al., it would have been obvious to one of ordinary 
skill in the art, at the time of the applicant's claimed invention, to additionally filter out 
"fault" faces from possible new faces, and the other possible new faces are verified to 
determine whether they are truly new faces by eigen-face analysis. The details will be 
omitted here. Please refer to the discussion above relating to claim 8. 
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49. The following is in regard to Claim 15. As shown above, the teachings of Yang et 
al., Haritaoglu et al., Kinjo, and Turk et al. can be combined so as to yield a face tracking 
system that adequately satisfies the limitations of claim 13. The limitations of claim 15 
were sufficiently addressed above with respect to claim 1 . See the discussion above 
relating to items (1.e.1) and (1.e.2). 

50. Claim 14 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Haritaoglu et al., Yang et al., Kinjo, and Turk, as applied to claim 1 1 , in further view of 
Lee et al. and Abdel-Mottaleb et al. 

51 . The following is in regard to Claim 14. As shown above with regard to claim 9, 
the teachings of Yang et al., Haritaoglu et al., Kinjo, and Turk et al., Lee et al. and Abdel- 
Mottaleb can be combined so as to yield a face tracking system, wherein face regions 
(i.e. "false" faces) are removed from consideration (i.e. "filtered out) if they fail to 
properly exhibit the following characteristics: 

(14.a.) Ratio between the height and width of a face (i.e. the aspect ratio - Kinjo 

column 1 1 , lines 32-35) 
(14.b.) Number of holes existing in a region (Kinjo column 14, lines 18-24). 
(14.c.) Convexity of a face (Kinjo, column 1 1 , lines 4-7). 
(14.d.) Compactness of a tracked region (i.e. the density of the grids - Lee et al. 

column 5, lines 37-40). 
(14.e) Statistical Variance of the face (Abdel-Mottaleb et al. column 4, lines 14- 

17). 

The details will not be repeated here. Please refer to the discussion above relating to 
Claim 9. 
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Allowable Subject Matter 

Objections. Allowable Subject Matter 

52. Claims 7 and 21 are objected to as being dependent upon a rejected base claim, 
but would be allowable if rewritten in independent form including all of the limitations of 
the base claim and any intervening claims. 

53. The following is a statement of reasons for the indication of allowable subject 
matter. 

54. The following is in regard to Claims 7 and 21. The criteria set forth in claims 7 
and 21 impose geometric constraints on detected face-like regions. Specifically, these 
criteria ensure that face-like regions have, using the Applicant's words, "two sharp down- 
edges in its right and left sides" (the first two equations of claims 7 and 21 are related to 
the slopes of these edges) and exhibit an appropriate aspect ratio. The criteria are 
generally related to the Applicant's detection of a so-called "protrusion shape". A 
multitude of prior art techniques were encountered that observe the aspect ratios of face 
regions to ensure their consistency with that of the human face. Furthermore, as 
discussed above, Kinjo disclose a face extraction method, wherein regions are 
determined to be face-like when they posses a shape similar to a protrusion shape. 
Eleftheriadis et al. (U.S. Patent 5,852,669), cited below, also make such a determination 
and, more particularly, investigate the "down-edges" of potential face regions 
(Eleftheriadis et al. Fig. 4). Despite this, no prior art face or head-tracking 
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method/systems were encountered that demonstrated, or even suggested, determining 
the existence of a face-like region based on the specific criteria set forth in claims 7 and 


55. The prior art made of record and not relied upon is considered pertinent to 
applicants disclosure: 

[1]-[13] generally relate to systems and methods that track at least one human face 
through a series of images. Most have clear similarities to the Applicant's claimed 
invention. 

[1] U.S. Patent 6,301,370. Steffens et al. . Publication Date. October 2001. 
Steffens et al.'s tracking method is particularly relevant to the Applicant's 
disclosure. In particular, Steffens et al., among other things, distinguishes 
between tracked and newly occurring faces, segments regions according to 
skin-color, analyzes silhouettes, uses motion to distinguish tracked regions, 
measures convexity of the detected faces and suggests the usage of 
eigenfaces to recognize faces as those that have been previously tracked. 

[2] U.S. Patent 6,754,389. Dimitrova et al. . Publication Date. June 2004. 

[3] U.S. Patent 6,665,446. Kato et al. . Publication Date. December 2003. 
Kato et al. track multiple faces. 

[4] U.S. Patent 5,912,980. Hunke et ai. Publication Date. June 1999. 

Hunke et al. track multiple faces. The method involves creating a difference 
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image. 

[5] U.S. Patent 6,639,998. Lee et al. . Publication Date. October 2003. 
[6] U.S. Patent Application Publication 2002/0000025. Darrell et al. . Publication 
Date. March 2001. 

Darrell et al. track faces by detecting a combination of motion, shape, and 
color in an image. 

[7] W*S: A Real-Time System for Detecting and Tracking People in 2-1/2 D. 

ECCV 1998. Haritaoglu et al. 
[8] Tracking Groups of People. Computer Vision and Image Understanding. 

80:42-56, 2000. McKenna et al. 

McKenna et al. disclose a tracking system, similar to that of Haritaoglu et al. 
McKenna et al. use an adaptive color model consisting of a Gaussian 
mixture. These types of models would inherently involve the statistical 
variance of the face (color). 
[9] Integrated Person Tracking Using Stereo, Color and Pattern Detection. IEEE 
1998. Darrell etal. 

[10] Pfinder: Real-Time Tracking of the Human Body. IEEE 1997. Wren et al. 
[11] Tracking of Multiple Faces for Human-Computer Interfaces and Virtual 
Environments. IEEE International Conference on Multimedia and Expo (III) 
2000. Huang et al. 
[12] A Real-Time Face Tracker. IEEE 1996. Yang et al. 
[1 3] Improved Tracking of Multiple Humans with Trajectory Prediction and 
Occlusion Modeling. IEEE 1998. Rosales et al. 
[14]-[15] relate to the detection of skin-color in an image and segmenting the image 
based on that detection. 
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[14] U.S. Patent Application Publication 2003/0179911. Ho et al. Publication 

Date: September 2003. 
[1 5] U.S. Patent Application Publication 2002/0159630. Buzuloiu et al. 
Publication Date: October 2002 
[16]-[18] relate to the detection of faces in an image based on geometric properties of an 
image. These methods detect a shape similar to the Applicant's "protrusion shape". 

[16] Locating Human Faces in Newspaper Photographs. IEEE 1989. Govindaraju 
et al. 

[17] Finding Face Features. Proceedings of the Second European Conference on 

Computer Vision 1992, Craw et al. 
[18] US. Patent 5,852,669. Eleftheriadis et al. Publication Date: December 1998. 


Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Kevin Siangchin whose telephone number is (703)305- 
7569. The examiner can normally be reached on 9:00am - 5:30pm, Monday - Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Amelia Au can be reached on (703)308-6604. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 
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